Learning from Crowds by Modeling Common Confusions

Crowdsourcing provides a practical way to obtain large amounts of labeled data at low cost. However, the annotation quality annotators varies considerably, which imposes new challenges in learning high-quality model from crowdsourced annotations. In this work, we provide perspective decompose noise into common and individual differentiate source confusion based on instance difficulty annotator expertise per-instance-annotator basis. We realize crowdsourcing by an end-to-end solution with two types adaptation layers: one is shared across capture their commonly confusions, other pertaining each confusion. To recognize annotation, use auxiliary network choose layers respect both instances annotators. Extensive experiments synthesized real-world benchmarks demonstrate effectiveness our proposed solution.